The present invention generally relates to a motion vector estimation circuit for estimating a motion vector for a macro block, which is a predetermined range in a frame or field of a moving picture. More specifically, the invention relates to a motion vector estimation circuit for accumulating the differences between the signal strength of the pixels of original and reference images corresponding to a motion vector candidate for the macro block, to select a motion vector candidate having a minimum accumulation result as a motion vector.
In a typical moving picture information processing, a motion vector is estimated by operating a value for a pixel in a macro block using the following technique. That is, for a certain motion vector candidate, the signal strength of a pixel in a macro block on an original image is compared with the signal strength of each of the pixels on a reference image which has a region corresponding to the macro block and which is obtained by the translating of the original image by a distance corresponding to a motion vector, to accumulate a value indicative of the difference between the signal strength for all of the pixels in the macro block. This operation is carried out for all of motion vector candidates, and the accumulated result obtained for each of the motion vector candidates is sequentially compared to estimate a candidate having a minimum accumulation result as a motion vector. In the above described technique, the term xe2x80x9creference imagexe2x80x9d means an image of a predetermined region corresponding to a macro block obtained by the translating of a macro block, which is a predetermined range in a frame or field of an original image, by a distance corresponding to a motion vector.
A motion vector estimation circuit for such a typical motion vector estimation is designed to carry out an accumulation processing at a high speed using a small amount of data by arranging a plurality of accumulator circuits in parallel and assigning the operation for a motion vector candidate to each of the accumulation circuits to output accumulation result data as a pipeline. In such a motion vector estimation circuit, each of the accumulation circuits is designed to carry out the operation of the signal-strength difference on the basis of the difference between the luminance value of a pixel forming a macro block of an original image and the luminance value of the corresponding pixel of a reference image. Specifically, each of the accumulator circuits carries out the operation for obtaining an absolute-value of the difference between luminance values of pixels of original and reference images, namely, |Luminance Value of Pixel of Original imagexe2x80x94Luminance Value of Pixel of Reference Image|.
Specifically, when the motion vector estimation is carried out by the above described pipeline architecture system, the accumulation for regions corresponding to the respective motion vectors is carried out, and the same estimation is carried out for all of predetermined motion vector candidates to select and adopt a motion vector candidate, which has a minimum (or maximum) estimated value, from the motion vector candidates as a motion vector for the macro block. If the number of motion vector candidates, which can be estimated at a time, is defined as a parallel number, the motion vector estimation is completed by repeating the accumulation times obtained by dividing the number of all of the motion vector candidates by the parallel number. Therefore, when the processing is carried out by a parallel number m, each m candidates are selected from the motion vector candidates, and this processing is repeated until all of the candidates are selected. For example, if the total number of the motion vector candidates is 1024 and if the number of columns thereof is 16, 64 accumulation operations are repeated to complete the motion vector estimation.
Therefore, the motion vector estimation is not completed unless the second through number n estimations are sequentially repeated after the first estimation of the accumulated value for the motion vector candidates 1 through m is carried out. Such a motion vector estimation circuit must always carry out the accumulation operations by a fixed number of cycles regardless of the quality of an input original image, so that there are problems in that the number of cycles unnecessary for estimating motion vectors is used, and electric power consumption necessary for driving a circuit for estimating motion vectors is increased. In order to solve such problems, the inventors of the present invention have already proposed a motion vector estimation system for stopping unnecessary accumulation operation in Japanese Patent Application No. 8-341928 (1996).
This conventional motion vector estimation system comprises the construction shown in FIG. 15, which is provided in a motion vector estimation system for selecting a motion vector candidate of a specific region in a macro block, which is an image of a predetermined range in a frame or field of an original image, to calculate the difference between the signal strength of pixels corresponding to the motion vector candidates of original and reference images, to accumulate the signal strength difference calculated for the pixels in the macro block to derive the accumulation results, to select a motion vector candidate having a local maximum or local minimum accumulation result, as a motion vector, from the motion vector candidates in the specific region in the macro block.
That is, in FIG. 15, the motion vector estimation system comprises: accumulating means 1 comprising a plurality of differential absolute-value accumulating circuits 2a, 2b through 2n, which are connected as a pipeline for parallel inputting original image data D1 and reference image data D2 for motion vector candidates to carry out the accumulation of a plurality of motion vectors in parallel; accumulation stopping means 10 having a separate stop circuit 11 for stopping the accumulation operation of the accumulating means 1 for each of the accumulating circuits 2a, 2b through 2n; and control means 5 for controlling the supply of the original image data D1 and the reference image data D2 for the motion vector candidates to the accumulating circuits 2a, 2b through 2n of the accumulating means 1 and for outputting a separate stop signal S3 for stopping the separate accumulation to the separate stop circuit 11 to control the stopping of the accumulation operation of the corresponding accumulating circuit 2a, 2b or 2n. 
Each of the accumulation results of the differential absolute-value accumulating circuits 2a, 2b through 2n forming the accumulating means 1 is outputted to a minimum-value information management circuit 4 as an accumulation signal S1. The minimum-value management circuit 4 compares the accumulation signal S1 with the minimum value of the differential absolute-values which have been inputted, and outputs a comparison result signal S2 to the control means 5. The control means 5 outputs a stop signal S3 to the accumulation stopping means 10 for controlling the stopping of the accumulation of each of the differential absolute-value accumulating circuits 2a through 2n forming the accumulating means 1 on the basis of the comparison result signal S2 supplied from the minimum-value information management circuit 4.
However, the conventional motion vector estimation system shown in FIG. 15 is not good from the point of view of the improvement efficiency. Because if the improvement efficiency is under the influence of the quality of the input moving picture to cause the reviewing order of a motion vector candidate, which is finally to be a motion vector, to be a later order in the whole motion vector candidates, the accumulation operation for that motion vector candidate is not carried out unless the accumulation operation for almost all of motion vector candidates at the substantially front of the reviewing order is carried out. For example, in a case where motion vector candidates are thus reviewed in fixed order, when the motion vectors on the screen are turned rearward as a whole, there are some possibilities of decreasing the motion vector finding efficiency for all of macro blocks.
If a motion vector candidate to be a final motion vector can be found at an initial stage of the estimation operation when a motion vector is estimated by the accumulation using the above described motion vector estimation system, the actual operation time for accumulation can be shorten to considerably increase the motion vector estimating efficiency and to reduce electric power consumption of each of arithmetic circuits for accumulating each of motion vector candidates in the estimation system.
It is therefore an object of the present invention to eliminate the aforementioned problems and to provide a motion vector estimation circuit which can improve operation time, i.e., operation efficiency, for motion vector estimation and which can reduce electric power consumption during retrieval operation.
In order to accomplish the aforementioned and other objects, according to one aspect of the present invention, there is provided a motion vector estimation circuit for setting, as motion vector candidates, vectors in an appropriate range for a macro block, which is an image of a predetermined range in a frame or field in a moving picture, to calculate a difference between signal strength for images corresponding to motion vector candidates of original and reference images, to accumulate the calculated difference between the signal strength for images in the macro block to derive accumulation results to select, as a motion vector, a motion vector candidate having a local maximum or local minimum accumulation result from the set motion vector candidates, the motion vector estimation circuit comprising: grouping means for dividing motion vector candidates in a search range, which is set for the motion vector candidates, into a plurality of motion vector candidate groups; priority order determining means for predicting an order from a motion vector candidate group, which has a high possibility of including a true motion vector, to a motion vector candidate group, which has a low possibility of including the true motion vector, to determine priority orders for accumulation to estimate motion vectors for each of motion vector candidate groups in accordance with the predicted order; accumulating means, comprising a plurality of accumulating circuits connected as a pipeline, for parallel inputting original and reference image data for the motion vector candidates of a plurality of motion vector candidate groups in accordance with the priority orders determined by the priority order determining means, to parallel process accumulation operations for a plurality of motion vectors; accumulation stopping means for separately stopping the accumulation operation for each of the accumulating circuits of the accumulating means; and control means for controlling supply of the original and reference image data for the motion vector candidates for each of the accumulating circuits, to output a separate stop signal for stopping accumulation in one of the accumulating circuits from the accumulation stopping means to stop accumulation in the one of the accumulating circuits, and for causing the accumulation stopping means to output the stop signal in an initial stage of a series of reviewing operations by causing the accumulating means to review the motion vector candidates of the group having the high possibility to complete the reviewing of the motion vector candidate having the high possibility serving as a final motion vector candidate in the initial stage of the series of reviewing operations.
The grouping means may be associated with the priority order determining means to form motion vector candidate group assigning means, the priority order determining means dynamically setting priority orders for each of the motion vector candidate groups, which are grouped by the grouping means, to accumulate the difference between signal strength of pixels of the original and reference images corresponding to motion vector candidates for each of the motion vector candidate groups in accordance with the priority orders.
The priority order determining means may include measuring means, incorporated in an image pick-up system, for measuring an angular speed or an angular acceleration for motion of a moving picture picked up by the image pick-up system, to estimate motion vectors in the moving picture using a reference motion vector derived on the basis of the measured value of the angular speed or angular acceleration supplied from the measuring means.
The priority order determining means may determine the priority orders for the motion vector candidate groups using, as a reference motion vector, a global motion vector of a target object in the moving picture, the global motion vector being obtained by searching the whole frame for a portion of a characteristic point of the target object.
The priority order determining means may determine the priority orders of motion vector candidate groups using, as a reference motion vector, a motion vector in a single macro block or in each of a plurality of macro blocks, which exist in the same frame as a frame including the macro block or macro blocks wherein motion vector estimation is carried out, and in which motion vector estimation has been carried out before the motion vector estimation in the macro block or macro blocks.
The priority order determining means may determine the priority orders of the motion vector candidate groups using, as a reference motion vector, a motion vector in a single macro block or in each of a plurality of macro blocks, in which motion vector estimation has been completed, the single macro block or the plurality of macro blocks being one or some of macro blocks, which are included in a frame at a point of time different from that of a frame including macro blocks wherein motion vectors are estimated, and which exist at coordinates in vicinity of the macro block or the same coordinates as those of the macro block.
The priority order determining means may include weighting means for suitably weighting a reference motion vector after selecting the reference motion vector, and determines priority orders for motion vector candidate groups weighted by the weighting means.
According to another aspect of the present invention, there is provided a motion vector estimation method for setting, as motion vector candidates, vectors in an appropriate range for a macro block, which is an image of a predetermined range in a frame or field in a moving picture, to calculate a difference between signal strength for images corresponding to motion vector candidates of original and reference images, to accumulate the calculated difference between the signal strength for images in the macro block to derive accumulation results to select, as a motion vector, a motion vector candidate having a local maximum or local minimum accumulation result from the set motion vector candidates, the motion vector estimation method comprising the steps of: grouping motion vector candidates in a search range, which is set for the motion vector candidates, into a plurality of motion vector candidate groups; predicting an order from a motion vector candidate group, which has a high possibility of including a true motion vector, to a motion vector candidate group, which has a low possibility of including the true motion vector, to determine priority orders for accumulation to estimate motion vectors in accordance with the predicted order; parallel inputting original and reference image data for the motion vector candidates, to parallel process accumulation operations for a plurality of motion vectors in accordance with the determined priority orders by means of a plurality of pipeline stages; selecting motion vectors, the number of which corresponds to the number of the plurality of pipeline stages; reviewing the motion vector candidates by means of the pipeline stages including the step of accumulation; determining whether all of the motion vector candidates included in the motion vector candidate group of that priority order have been reviewed; determining whether motion vectors for all of the motion vector candidate groups have been reviewed in accordance with the priority orders; and outputting a remaining motion vector as a true motion vector.
The step of grouping the motion vector candidates into the plurality of motion vector candidate groups may be carried out using, as a reference motion vector, a global motion vector of a target object obtained by searching the whole frame for a portion serving as a characteristic point of the target object in the moving picture.
The step of determining the priority orders of the plurality of motion vector candidate groups may include a step of suitably weighting a reference motion vector after selecting the reference motion vector.
With the above construction, according to the present invention, motion vector candidates to be reviewed are divided into groups of predetermined regions, and the order for reviewing motion vectors is suitably changed every group, so that it is possible to prevent macro blocks having lowered finding efficiency in a frame from being sequentially arranged. Specifically, the priority orders of motion vector candidates to be reviewed are assigned on the basis of the following standard. When macro blocks in a certain frame is noticed:
(1) Priority is given to a motion vector candidate group including a motion vector of a macro block arranged in vicinity of a macro block, in which motion vector estimation has been carried out, in the same frame, or a motion vector candidate group arranged in vicinity thereof.
(2) Priority is given to a motion vector candidate group including a motion vector of a macro block or a macro block adjacent thereto, in which motion vector estimation has been completed, in temporally last and next frames, or a motion vector candidate group in vicinity thereof.
(3) When a motion vector estimation circuit is incorporated in a hand camera or the like, there are some cases where a device, such as an optical fiber gyro, which is capable of acquiring an angular speed or angular acceleration information, is mounted therein in order to prevent the movement of the hands. Priority is given to a motion vector candidate group including a motion vector based on the information from such a device, or a motion vector candidate group arranged in vicinity thereof.
(4) Priority is given to a motion vector candidate group including a global motion vector of a target object obtained by searching for a portion serving as a characteristic point of the target object, or a motion vector candidate group arranged in vicinity thereof.
By these techniques, it is possible to statistically improve the order for reviewing a vector to be finally a motion vector. By such improvement, it is possible to enhance the efficiency of the motion vector estimation circuit. Furthermore, the motion vector estimation circuit and the motion vector estimation method according to the present invention includes grouping means and priority order determining means in addition to the accumulating means and accumulation stopping means in the motion vector estimation system and method disclosed in Japanese Patent Application No. 8-341928 (1996).